92. (Amended)\The computer program product according to claim 80, 
wherein said digital image is stored iii a database of digital images and wherein said digital image 
can be retrieved using a icon represent ng a stereotype. 

93. (Amended) 1 he computer program product according to claim 80, 
wherein said digital image is stored in a database of digital images and wherein said digital image 
can be retrieved from said database us; ng either a keyword or icon representing a generalisation, 
or broader version, of a stereotype. I 

1 — - — - 

REMARKS 

This application has been reviewed in light of the Office Action dated May 24, 
2002. Claims 1, 3-18, 20-35, 37-52, 54-66, 68-80, and 82-93 are presented for examination. 
Claims 2, 19, 36, 53, 67, and 81 have been canceled, without prejudice or disclaimer of the 
subject matter presented therein. Claims 1, 3-8, 18, 20-25, 27, 30, 34, 35, 37-42, 52, 54-60, 62- 
66, 68-74, 76-80, 82-88, and 90-93 have been amended to define more clearly what Applicant 
regards as her invention. Claims 1, 18, 35, 52, 66 and 80 are in independent form. Favorable 
reconsideration is requested. 

Claims 4-9 were objected to as to informalities noted in paragraph 1 of the 

Office Action. 

Applicant has carefully reviewed and amended Claims 4-9 to overcome the 
noted objections. It is believed that the objections to Claims 4-9 have been remedied, and 
withdrawal is therefore respectfully requested. 
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Claims 1-10, 18-27, 35-44, 52-61, 66-75, and 80-89 were rejected under 35 
U.S.C. § 102(b) as being anticipated by "A Markov Random Field Model-Based Approach To 
Image Interpretation, IEEE Paper ISBN: 0162-8828 {Modestino et al). 

Claims 1 1, 17, 28, 34, 45, 51, 62-65, 76-79, and 90-93 were rejected under 35 
U.S.C. § 103(a) as being unpatentable over Modestino et al in view of U.S. Patent No. 5,930,783 
(Li et al). Claims 12-16, 29-33, and 46-50 were rejected under 35 U.S.C. § 103(a) as being 
unpatentable over Modestino et al. further in view of Li et al and U.S. Patent No. 6,360,234 
(Jain et al). 

First, cancellation of Claims 2, 19, 36, 53, 67, and 81 renders their rejections 

moot. 

As shown above, Applicant has amended independent Claims 1, 18, 35, 52, 66, 
and 80 in terms that more clearly define the present invention. Applicant submits that these" 
amended independent claims, together with the remaining claims dependent thereon, are 
patentably distinct from the cited prior art for at least the following reasons. 

The aspect of the present invention set forth in Claim 1 is a method of 
classifying a digital image. The method comprises the steps of segmenting the image into 
substantially homogeneous regions and processing the regions to provide a region adjacency 
graph for the digital image. The region adjacency graph represents adjacencies between the 
regions. The method also comprises the steps of analyzing the region adjacency graph for 
predetermined patterns of regions, and classifying the digital image as one of a plurality of 
stereotypes according to each identified pattern. 
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One important feature of Claim 1 is that the method classifies the digital image 
as one of a plurality of stereotypes according to each identified pattern. 

Modestino et al, as understood by Applicant, discloses an image interpretation 
method based on a Markov Random Field (MRF) model. Apparently, in the Modestino et al 
method, an image is segemented into a collection of disjoint regeions that form the nodes of an 
adjacency graph. Object labels are assigned to the segmented regions using domain knowledge, 
extracted feature measurements, and spatial relationships between the various regions. The 
interpretation labels are modeled as an MRF on the corresponding adjacency graphs, and the 
image interpretation problem is then formulated as a maximum posteriori (MAP) estimation rule, 
given the domain knowledge and region based measurements. The domain knowledge is 
represented as clique functions. Simulated annealing is used to find the best realization or % 
optimal MAP interpretation. 

In making the rejection of dependent Claim 8, under 35 U.S.C. § 102(b), the 
Office Action states that Modestino et al discloses that the stereotypes are represented in a 
hierarchical arrangement. However, Applicant submits that Modestino et al in general and 
particularly at Fig. 1 and Table IQ (b) and IV (b), fails to disclose, teach or suggest the method as 
recited in Claim 1, of classifying a digital image as one of a plurality of stereotypes according to 
each identified pattern. 

As described at page 14, lines 13 and 14 of the present specification, a 
stereotype is a higher level expression which can be used for the purposes of retrieval on a more 
generic basis than is provided by the use of semantic labels. Further, as described at page 15, 
lines 12-19, the generated stereotypes can form the basis of an image retrieval system where 
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images can be retrieved on the basis of stereotype name matching. The icons can be presented in 
the form of icon trees through which the user can navigate, where the icon trees represent the 
hierarchical arrangement of stereotypes. The user can select the icon representing the stereotype 
of the desired image(s), and a query can be generated for the request. As also described at page 
16, lines 2-6, the stereotype icons provide a convenient means for formulating a query to an 
image (or video) database. Still further, the image retrieval system can be implemented without 
the costly requirement of manual annotation, since the stereotypes are automatically generated by 
a digital image or digital video interpretation system. 

Accordingly, Applicant submits that Claim 1 is not anticipated by Modestino et 
al, and respectfully request withdrawal of the rejection under 35 U.S.C. § 102(b). 

Li et aL, as understood by Applicant, relates to a computer implemented 
method for searching and retrieving images contained within a database of images in whicfcboth 
semantic and cognitive methodologies are utilized. The method accepts a semantic and cognitive 
description of an image to be searched from a user, and successively refines the search utilizing 
semantic and cognitive methodologies and then ranks the results for presentation to the user. 
Similarly, Jain et al 9 as understood by Applicant, relates to a system and method for video 
cataloguing. The video is catalogued according to predefined user definable metadata. The 
metadata of Jain et al is used to index and then retrieve encoded video. 

In many cases users do not wish to retrieve images using specific constraints as 
described in Li et al (e.g. finding all images where a person is to the left of a control tower). 
This is particularly true for graphic designers who are looking for an image to fit into a design or 
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presentation. In such situations, the present invention, as defined in Claim 1, can act as a filter, 
so that the designer can select the best image by browsing. 

Applicant submits that neither Li et aL, Jain et al, nor any combination 
thereof, added to the disclosure of Modestino et al, discloses or suggests the method as recited in 
Claim 1, of classifying a digital image as one of a plurality of stereotypes according to each 
identified pattern. 

Accordingly, Applicant submits that Claim 1 is believed to be in condition for 

allowance. 

Independent Claims 18, 35, 52, 66, and 80 include the similar feature of 
classifying the digital image as one of a plurality of stereotypes according to each identified 
pattern, as discussed above in connection with Claim 1. Accordingly, Claims 18, 35, 52, 66, and 
80 are believed to be patentable for reasons substantially similar to those discussed above in 
connection with Claim 1 . 

A review of the other art of record has failed to reveal anything which, in 
Applicant's opinion, would remedy the deficiencies of the art discussed above, as references 
against the independent claims herein. These claims are therefore believed patentable over the 
art of record. 

The other claims in this application are each dependent from one or another of 
the independent claims discussed above and are therefore believed patentable for the same 
reasons. Since each dependent claim is also deemed to define an additional aspect of the 
invention, individual reconsideration of the patentability of each on its own merits is respectfully 
requested. 
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In view of the foregoing amendments and remarks, Applicant respectfully 



requests favorable reconsideration and early passage to issue of the present application. 



Applicant's undersigned attorney may be reached in our New York office by 



telephone at (212) 218-2100. All correspondence should continue to be directed to our below 
listed address. 



FITZPATRICK, CELLA, HARPER & SCINTO 

30 Rockefeller Plaza 

New York, New York 101 12-3801 

Facsimile: (212) 218-2200 

NYMAIN 299932 



Respectfully submitted, 




rftofney for Applicant 
.egistration No. 38,586 
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VERSION WITH MARKINGS TO SHOW CHANGES MADE TO CLAIMS 

1 . (Amended) A method of classifying a digital image, said method 
comprising the steps of: 



[representing] for the digital image , said region adjacency graph representing adjacencies 
between said regions ; [and] 

analysing said region adjacency graph for predetermined patterns [and for 
each identified pattern selecting a classification of said digital image] of regions; and 

classifying said digital image as one of a plurality of stereotypes according 
to each identified pattern . 



segmenting said image into substantially homogeneous regions; 



[providing] processing said regions to provide a region adjacency graph 



Claim 2 has been canceled. 



3. (Amended) The method according to claim [2] 1, wherein [a stereotype is 
assigned to a region adjacency graph] said digital image is classified on the basis of a size of one 
or more regions of said digital image. 





4. 



(Amended) The method according to [any one of] claim 3, wherein [a 



stereotype is assigned to a region adjacency graph] said digital image is classified on the basis of 
an adjacency of said regions. 

5. (Amended) The method according to [anyone of] claim [4] I, wherein [a 
stereotype is assigned to a region adjacency graph] said digital image is classified on the basis of 
[its] semantic label content of said region adjacency graph . 



stereotype is assigned to a region adjacency graph] said digital image is classified on the basis of 
a mean colour of one or more regions of said region adjacency graph . * 



6. 



(Amended) The method according to [anyone of] claim [4] 1, wherein [a 



7. 



(Amended) The method according to [anyone of] claim [6] 1, wherein said 



plurality of stereotypes are stored in an association lookup table. 



8. 



(Amended) The method according to claim [6] I, wherein said stereotypes 



are represented in a hierarchal arrangement. 



18. (Amended) An apparatus for classifying a digital image, said apparatus 



composing: 



segmenting means for segmenting said image into substantially 



homogeneous regions; 



[providing means for providing] processing means for processing said 
regions to provide a region adjacency graph [representing] for the digital image , said region 
adjacency graph representing adjacencies between said regions ; [and] 

analysing means for analysing said region adjacency graph for 
predetermined patterns [and for each identified pattern selecting a classification of said digital 
image] of regions; and 

classification means for classifying said digital image as one of a plurality 
of stereotypes according to each identified pattern . 

Claim 1 9 has been canceled. 

20. (Amended) The apparatus according to claim [19] 18, wherein [a 
stereotype is assigned to a region adjacency graph] said digital image is classified on the basis of 
a size of one or more regions of said digital image. 

21 . (Amended) The apparatus according to claim 20, wherein [a stereotype is 
assigned to a region adjacency graph] said digital image is classified on the basis of an adjacency 
of said regions. 

22. (Amended) The apparatus according to claim [21] 18, wherein [a 
stereotype is assigned to a region adjacency graph] said digital image is classified on the basis of 
[its] semantic label content of said region adjacency graph . 
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23. (Amended) The apparatus according to claim [21] 18, wherein [a 
stereotype is assigned to a region adjacency graph] said digital image is classified on the basis of 
a mean colour of one or more regions of said digital image . 

24. (Amended) The apparatus according to claim [23] 18, wherein said 
plurality of stereotypes are stored in an association lookup table. 

25. (Amended) The apparatus according to claim [23] 18, wherein said 
stereotypes are represented in a hierarchal arrangement. 

27. (Amended) The apparatus according to claim [26] 18, wherein said region 
adjacency graph is provided by analysing contextual data associated with one or more regions of 
said digital image. 

30. (Amended) The apparatus according to claim [29] 18, further comprising 
metadata providing means for providing metadata associated with each digital image, wherein 
said metadata includes said stereotypes of each digital image. 

34. (Amended) The apparatus according to Claim [33] 18, which said digital 
image is stored in a database of digital images and wherein said classification can be used to 
retrieve said digital image from said database. 
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35. (Amended) A computer program product comprising a computer readable 
medium having a computer program recorded for classifying a digital image, said computer 
program product comprising: 

segmenting module for segmenting said image into substantially 

homogeneous regions; 

[providing module for providing] processing module for processing said 
regions to provide a region adjacency graph [representing] for the digital image , said region 
adjacency graph representing adjacencies between said regions ; [and] 

analysing module for analysing said region adjacency graph for 
predetermined patterns [and for each identified pattern selecting a classification of said digital 
image] of regions; and 

classification module for classifying said digital image as one of a plurality 
of stereotypes according to each identified pattern . 

Claim 36 has been canceled. 



37. (Amended) The computer program product according to claim [36] 35> 
wherein [a stereotype is assigned to a region adjacency graph] said digital image is classified on 
the basis of a size of one or more regions of said digital image. 



38. (Amended) The computer program product according to claim 37, wherein 
[a stereotype is assigned to a region adjacency graph] said digital image is classified on the basis 
of an adjacency of said regions. 

39. (Amended) The computer program product according to claim [38] 35, 
wherein [a stereotype is assigned to a region adjacency graph] said digital image is classified on 
the basis of [its] semantic label content of said region adjacency graph . 

40. (Amended) The computer program product according to claim [38] 35 , 
wherein [a stereotype is assigned to a region adjacency graph] said digital image is classified on 
the basis of a mean colour of one or more regions of said digital image . ^ 

41 . (Amended) The computer program product according to claim [40] 35, 
wherein said plurality of stereotypes are stored in an association lookup table. -•■ 

42. (Amended) The computer program product according to claim [41] 35, 
wherein said stereotypes are represented in a hierarchal arrangement. 

52. (Amended) A method of classifying a digital image signal, said method 
comprising the steps of: 

segmenting said image into substantially homogeneous regions; 
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[providing a labelled] processing said regions to provide a labelled region 
adjacency [grpah representing at least part of] graph [the digital image signal] comprising at least 
one semantic label and representing at least part of the digital image siganl ; 

providing a plurality of stereotype classifications, for each of a plurality of 
patterns, wherein each said pattern comprises: 

(i) a set of labelled regions; or 

(ii) a set of labelled regions and corresponding adjacency 

information; 

analysing said labelled region adjacency graph for the presence of 
predetermined patterns; and 

for each pattern identified, selecting from said plurality of classifications a 
stereotype classification for the digital image. 

Claim 53 has been canceled. 

54. (Amended) The method according to claim [53] 52, wherein [a stereotype 
is assigned to the digital image signal] said digital image is classified on the basis of [the] 
semantic label content of [one or more regions in the labelled] said region adjacency graph. 

55. (Amended) The method according to claim [53] 52, wherein a stereotype 
is assigned to the digital image signal on the basis of the adjacency of a set of regions with 
specified labels in the labelled region adjacency graph. 
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56. (Amended) The method according to claim [53] 52, wherein a stereotype 
is assigned to the digital image signal on the basis of the size of one or more regions with a 
specified label in the labelled region adjacency graph. 

57. (Amended) The method according to claim [53] 52, wherein a stereotype 
is assigned to the digital image signal on the basis of a label which represents the mean colour of 
one or more regions in the labelled region adjacency graph. 

58. (Amended) The method according to claim [53] 52, wherein a stereotype 
is assigned to the digital image signal on the basis of a label which represents the mean colour 
texture of one or more regions in the labelled region adjacency graph. X 

59. (Amended) The method according to claim [58] 52, wherein said plurality 
of stereotypes are stored in an association lookup table. 

60. (Amended) The method according to claim [59] 52, wherein said 
stereotypes are represented in an hierarchical arrangement. 

62. (Amended) The method according to claims [61] 52, wherein each of said 
stereotypes is represented by one of a plurality of icons. 
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63. (Amended) The method according to claims [62] 52, where said digital 
image is stored in a database of digital images and wherein said digital image can be retrieved 
from said database using a keyword representing a stereotype. 

64. (Amended) The method according to claim [62] 52, where said digital 
image is stored in a database of digital images and wherein said digital image can be retrieved 
from said database using an icon representing a stereotype. 

65. (Amended) The method according to claim [62] 52, where said digital 
image is stored in a database of digital images and wherein said digital image can be retrieved 
from said database using either a keyword or icon representing a generalisation, or broader*-, 
version, of a stereotype. 



66. (Amended) An apparatus for classifying a digital image signal, said •* 
apparatus comprising: 

segmenting means for segmenting said image into substantially 

homogeneous regions; 

[region adjacency graph providing means for providing a labelled region 
adjacency graph] processing means for processing said regions to provide a labelled region 
adjacency graph comprising at least one semantic label and representing at least part of the digital 
image signal; 
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classification providing means for providing a plurality of stereotype 



classifications, for each of a plurality of patterns, wherein each said pattern comprises: 



(i) 



a set of labelled regions; or 



(ii) a set of labelled regions and corresponding adjacency 



information; and 



analysing means for analysing said labelled region adjacency graph for the 



presence of predetermined patterns, wherein for each pattern identified, said classification 
providing means provides a stereotype classification for the digital image selecting from said 
plurality of stereotype classifications. 



68. (Amended) The apparatus according to claim [67] 66, wherein a stereotype 
is assigned to the digital image signal on the basis of the semantic label content of one or more 
regions in the labelled region adjacency graph. 

69. (Amended) The apparatus according to claim [67] 66, wherein a stereotype 
is assigned to the digital image signal on the basis of the adjacency of a set of regions with 
specified labels in the labelled region adjacency graph. 



Claim 67 has been canceled. 
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70. (Amended) The apparatus according to claim [67] 66, wherein a stereotype 
is assigned to the digital image signal on the basis of the size of one or more regions with a 
specified label in the labelled region adjacency graph. 

71. (Amended) The apparatus according to claim [67] 66, wherein a stereotype 
is assigned to the digital image signal on the basis of a label which represents the mean colour of 
one or more regions in the labelled region adjacency graph. 

72. (Amended) The apparatus according to claim [67] 66, wherein a stereotype 
is assigned to the digital image signal on the basis of a label which represents the mean colour 

texture of one or more regions in the labelled region adjacency graph. ^ * 

73. (Amended) The apparatus according to claim [72] 66, wherein said r 

plurality of stereotypes are stored in an association lookup table. ' ^ 

74. (Amended) The apparatus according to claim [73] 66, wherein said 
stereotypes are represented in an hierarchical arrangement. 

76. (Amended) The apparatus according to claim [75] 66, wherein each of said 
stereotypes is represented by one of a plurality of icons. 
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77. (Amended) The apparatus according to claim [76] 66, where said digital 
image is stored in a database of digital images and wherein said digital image can be retrieved 
from said database using a keyword representing a stereotype. 

78. (Amended) The apparatus according to claim [76] 66, where said digital 
image is stored in a database of digital images and wherein said digital image can be retrieved 
from said database using a icon representing a stereotype. 

79. (Amended) The apparatus according to claim [76] 66, where said digital 
image is stored in a database of digital images and wherein said image can be retrieved from said 
database using either a keyword or icon representing a generalisation, or broader version, of a 
stereotype. 

80. (Amended) A computer program product comprising a computer readable 
medium having a computer program recorded for classifying a digital image signal, said . 
computer program product comprising: 

segmenting module for segmenting said image into substantially 

homogenous regions; 

processing module for processing said regions to provide a labelled region 
adjacency graph [providing module for providing a labelled region adjacency graph] comprising 
at least one semantic label and representing at least part of the digital image signal; 
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classification providing module for providing a plurality of stereotype 
classifications, for each of a plurality of patterns, wherein each said pattern comprises: 

(i) a set of labelled regions; or 

(ii) a set of labelled regions and corresponding adjacency 

information; and 

analysing module for analysing said labelled region adjacency graph for 
the presence of predetermined patterns, wherein for each pattern identified, said classification 
providing module provides a stereotype classification for the digital image selecting from said 
plurality of stereotype classifications. 



Claim 81 has been canceled. 



82. (Amended) The computer program product according to claim [81] 80 , 
wherein a stereotype is assigned to the digital image signal on the basis of the semantic label 
content of one or more regions in the labelled region adjacency graph. 

83. (Amended) The computer program product according to claim [81] 80, 
wherein a stereotype is assigned to the digital image signal on the basis of the adjacency of a set 
of regions with specified labels in the labelled region adjacency graph. 
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84. (Amended) The computer program product according to claim [81] 80, 
wherein a stereotype is assigned to the digital image signal on the basis of the size of one or more 
regions with a specified label in the labelled region adjacency graph. 

85 . (Amended) The computer program product according to claim [81] 80, 
wherein a stereotype is assigned to the digital image signal on the basis of a label which 
represents the mean colour of one or more regions in the labelled region adjacency graph. 

86. (Amended) The computer program product according to claim [81] 80, 
wherein a stereotype is assigned to the digital image signal on the basis of a label which 
represents the mean colour texture of one or more regions in the labelled region adjacency^graph. 

87. (Amended) The computer program product according to claim [86] 80, 
wherein said plurality of stereotypes are stored in an association lookup table. ,i 

88. (Amended) The computer program product according to claim [87] 80, 
wherein said stereotypes are represented in an hierarchical arrangement. 

90. (Amended) The computer program product according to claim [81] 80 , 
wherein each of said stereotypes is represented by one of a plurality of icons. 
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91 . (Amended) The computer program product according to claim [90] 80, 
wherein said digital image is stored in a database of digital images and wherein said digital image 
can be retrieved using a keyword representing a stereotype. 

92. (Amended) The computer program product according to claim [90] 80, 
wherein said digital image is stored in a database of digital images and wherein said digital image 
can be retrieved using a icon representing a stereotype. 

93. (Amended) The computer program product according to claim [90] 80, 
wherein said digital image is stored in a database of digital images and wherein said digital image 
can be retrieved from said database using either a keyword or icon representing a generalisation, 
or broader version, of a stereotype. 
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